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EXECUTIVE  SUMMARY 


The  Department  of  Defense  continues  to  emphasize  the  importance  of 


meaningful  tests  and  evaluations  of  new  weapon  systems  early  in  the  system 


acquisition  process.  A key  element  in  a weapon  system's  development  test 


and  evaluation  and  the  operational  suitability  portion  of  its  operational 


test  and  evaluation  is  an  evaluation  of  its  qualitative  maintainability. 


The  purpose  of  this  study  project  is  to  identify  evaluation  factors 


to  consider  when  planning  and  conducting  an  Air  Force  aircraft  qualitative 


maintainability  test  and  evaluation  at  a test  site.  A total  of  44 


significant  evaluation  factors  are  Identified.  Fifteen  of  the  factors 


are  evaluation  factors  one  should  consider  when  "planning  and  conducting" 


an  evaluation.  Twenty>nlne  of  the  factors  are  evaluation  factors  one  should 


consider  for  actually  "evaluating"  the  aircraft  weapon  system.  The  most 


important  aspect  regarding  evaluation  factors  to  consider  is  to  identify 


them  early  and  plan  the  evaluation  program  effectively  using  the  factors 


as  guides.  Once  the  program  is  planned,  evaluation  managers  must  insure 


evaluation  personnel  are  fully  aware  of  the  factors  to  be  evaluated  and 


can  relate  their  findings  and  evaluation  results  to  the  qualitative  main- 


tainability of  the  aircraft  weapon  system  in  the  operational  environment. 


The  report  is  Intended  to  serve  as  an  aid  and  a source  of  information 


for  personnel  responsible  for  planning  and  conducting  qualitative  maintain- 


ability test  and  evaluation  programs  at  test  sites. 
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SECTION  I 


INTRODUCTION 


irpoae  oi 


In  a statement  by  Dr.  Malcoim  R.  Currie,  former  Director  of  Defense 
Research  and  Engineering,  to  the  Congress  of  the  United  States,  94th 
Congress,  Second  Session,  3 February  1976,  on  The  Department  of  Defense 
Program  of  Research.  Development.  Test  and  Evaluation.  FY  1977;  Dr.  Currie 
stated: 


Test  and  Evaluation  (T&E)  provides  the  essential  Informa- 
tion required  for  effective  management  of  the  system  acquisition 
process.  We  cannot  make  good  decisions  on  the  acquisition  of 
systems  without  objective  systems  performance  data.  For  this 
reason,  I will  continue  to  emphasize  a strong,  Independent  DOD 
T6eE  program.  Our  current  T&E  program  has  the  following  major 
objectives: 

Continue  to  improve  our  capability  for  providing  independent 
and  objective  T&E  Inputs  to  the  weapon  system  acquisition  process 

Conduct  adequate  Development  Test  and  Evaluation  (DX&E)  to 
verify  that  engineering  designs  are  in  hand  and  adequate 
Operational  Test  and  Evaluation  (OT&E)  to  verify  operational 
effectiveness  and  suitability  of  new  systems 

The  past  few  years  have  seen  significant  improvements  in 
DOD  procedures  for  conducting  T&E  of  major  weapon  systems. 

High  level  emphasis  on  T&E  was  assured  Initially  by  creation  of 
the  Office  of  Deputy  Director  of  Defense  Research  and  Engineering 
(Test  and  Evaluation) .... 

Further  strengthening  of  DOD  T&E  has  been  achieved  by  the 
establishment  of  independent  operational  T&E  agencies  within 
each  of  the  Services ... .These  agencies  plan  and  conduct  tests 
and  submit  their  independent  evaluations  directly  to  the  Chiefs 
of  their  respective  Services.... 

The  need  for  adequate  T&E  has  been  emphasized  by  the 
publication  of  appropriate  DOD  Directives.  Specificallj^  DOD 
Directive  5000.3  clearly  defines  the  role  of  T&E  In  the 
acquisition  of  major  weapon  systems.  Other  directives  dealing 
with  particular  aspects  of  the  weapons  acquisition  process 
include  specific  reference  to  required  T&E  procedures. 
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Additionally,  the  Individual  Services  have  published  their  own 
directives  for  Implementation  of  DOD  policies  on  T&G. 

Test  and  evaluation  continues  to  play  a key  role  In  the 
proceedings  of  the  DSARC,  which  meets  at  each  Important  milestone  In 
a major  weapon  system  acquisition  program  to  consider  whether  the 
program  should  be  advanced  to  ite  next  phase.  Prior  to  each 
DSARC  meeting,  the  Deputy  Director  (Test  and  Evaluation)  submits 
to  each  DSARC  principal  his  Independent  evaluation  of  the  adequacy 
of  testing  conducted  to  date  and  of  testing  planned  for  the  future. 

He  actively  partlclpatec  In  DSARC  discussions  leading  to 
recommendations  on  whether  program  advancement  Is  warranted. 

Finally,  after  a DSARC  review  is  completed,  he  submits  directly 
to  the  Secretary  of  Defense  his  Independent  assessment  of  a 
program's  T&E  status  for  consideration  along  with  the  recommenda- 
tions of  the  DSARC.  In  1975  the  DD(T£cE)  participated  In  30  DSARC 
reviews  of  27  different  major  programs. 

In  addition  to  periodic  review  by  the  DSARC,  the  T&E  status 
of  each  major  acquisition  program  is  subject  to  a continuing  review 
by  the  DD  (T&E)  and  his  star*..  An  important  tool  used  in  con- 
ducting these  continuing  reviews  is  the  Test  and  Evaluation  Master 
Plan  (i’HMP)  which,  in  accordance  with  DODD  5000.3,1s  required  for 
each  major  program.  The  TEMP  provides  an  overall  description  of 
objectives,  methodology  and  schedules  for  T&E  to  be  conducted 
throughout  the  life  cycle  of  a weapon  system.  Through  a continual 
process  of  review  and  updating,  the  TEMP  is  made  to  accurately  reflect 
current  T&E  requirements  for  a program.  A significant  benefit  of 
the  preparation  of  TEMPs  is  the  early  establishment  of  a close 
working  relationship  between  the  development  and  T&E  agencies  In 
veviewing  TEMPs  the  DD  (T&E)  is  especially  concerned  that  they 
Include  meaningful,  specifically  defined  technical  and  operational 
performance  objectives  against  which  program  progress  can  be 
accurately  measured ... . (30:X-1)1 

Dr.  Currie's  comments  clearly  Indicate  the  continuing  enphasis  the 
DOD  is  placing  on  realistic,  effective,  and  meaningful  test  and  evaluations 
of  new  weapon  systems.  A recent  January  18,  1977  reissue  of  DOD  Directive 
5000.1  "Major  System  Acquisitions",  reemphasized  the  DOD's  clear  and 
definite  position  regarding  the  Importance  of  test  and  evaluation 
commencing  as  early  as  possible  in  the  major  systems  acquisition  process. 

^This  notation  will  be  used  throughout  the  report  for  sources  of  quota- 
tions and  major  references.  The  first  number  is  the  source  listed  in  the 
bibliography.  The  second  number  is  the  page  in  the  reference. 
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It  states  an  estimate  of  the  military  utility  and  of  the  operational 
effectiveness  and  operational  suitability  including  loplstlcs  support 
requirements  vlll  be  accomplished  prior  to  making  a commitment  to  large- 
scale  production.  As  realistic  a test  environment  as  possible  and  an 
acceptable  representation  of  the  future  operational  system  will  be  used  In 
the  test  and  evaluation  program  (12:8). 

As  Dr.  Currie  Indicated  In  his  statement  to  Congress,  the  TEMP  is  an 
Important  tool  for  planning  and  conducting  an  effective  test  and  evaluation 
program.  A key  element  In  a weapon  system's  DT&E  and  the  operational 
suitability  portion  of  Its  OT&E  is  an  evaluatl«m  of  its  qualitative  main- 
tainabllity.  Planning  factors  to  acconmiodate  the  eva'  latior.  of  qualitative 
maintainability  and  the  provisions  for  conducting  the  evaluation  are 
included  in  the  TEMP. 

The  purpose  of  this  study  project  is  to  identify  and  understa.vl  the 
significant  evaluation  factors  which  should  be  considered  when  planning  and 
conducting  an  Air  Force  qualitative  maintainability  test  and  evaluation  at 
a test  site  in  order  to  provide  meaningful  recommendations  for  personnel 
responsible  for  preparing  the  qualitative  maintainability  evaluation 
portion  of  test  plans  and  conducting  such  testa  ,:nd  evaluations. 

Specific  Goals  of  the  Study  Project 

The  specific  goals  of  the  study  project  are  to  identify  significant 
evaluation  factors  which  are  fundamental  to  planning  and  conducting  an  Air 
Force  aircraft  qualitative  maintainability  test  and  evaluation  at  a test 
site  and  to  explore  the  scope  of  the.  evaluation  that  can  be  realistically 
and  meaningfully  accomplished  lu  a test  and  evaluation  environment  at  a 
test  site  by  identifying  the  factors. 
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Definitions 


The  reader  is  assumed  to  be  reasonably  familiar  with  the  terms  used 
in  this  report.  Terms  which  are  unique  to  the  study  project  are  defined 
within  the  text  of  the  report. 

Scope  of  the  Study  Project 

It  became  readily  apparent  to  the  author  when  selecting  his  study 
project  that  it  would  be  necessary  to  define  the  scope  of  the  study  in 
sufficient  detail  to  permit  the  reader  to  fully  understand  its  limits. 
Volumes  of  material  could  be  written  on  the  general  subject  of  aircraft 
maintainability  test  and  evaluation.  However,  the  author  elected  to 
investigate  a very  limited  aspect  of  this  broad  subject.  The  study  speci- 
fically pertains  to  qualitative  maintainability  test  and  evaluation  factors 
for  United  States  Air  Force  aircraft.  The  evaluation  is  that  which  can  be 
accomplished  by  maintenance  personnel  physically  located  at  a test  site. 
Interfaces  with  personnel  and  activities  beyond  the  geographical  boundaries 
of  the  test  site,  which  Impact  on  the  test  site  evaluation,  are  discussed 
where  appropriate.  The  qualitative  maintainability  evaluation  is  accom- 
plished within  the  constraints  of  a DT&E  and  Initial  Operational  Test  and 
Evaluation  (lOT&E)  environment.  The  study  project  does  not  address 
follow-on  OT&E  accomplished  subsequent  to  a favorable  DSARC  Milestone  III 
Production  and  Deployment  Decision.  Only  those  evaluation  factors  which 
are  considered  to  be  significant  ones  which  should  be  c'  ..^idered  when 
planning  and  conducting  the  qualitative  maintainability  test  and  evaluation 
are  investigated. 


Organisation  of  the  Report 

Section  I Includes  a statement  of  the  purpose  of  the  study  project, 
the  specific  goals  of  the  project,  definitions,  the  scope  of  the  study 
project,  and  the  organisation  of  the  report. 

Section  II  contains  a discussion  of  the  evaluation  factors  which 
define  the  environment  in  which  the  qualitative  maintainability  test  and 
evaluation  is  conducted  and  identifies  all  individual  evaluation  factors 
to  consider  when  planning  and  conducting  an  Air  Force  aircraft  qualitative 
maintainability  test  and  evaluation  at  a test  site. 

Section  III  includes  an  analysis  and  evaluation  of  the  individual 

evaluation  factors. 

Section  IV  presents  conslusions,  recoimnendations,  and  implications 
arrived  at  by  the  author  based  upon  the  study  effort  documented  in  this 
report. 


5 


SECTION  II 


EVALUATION  FACTORS  TO  BE  CONSIDERED 
The  Environment 

There  are  numerous  factors  which  define  the  environmeniL  In  which  an 
Air  Force  aircraft  qualitative  maintainability  test  and  evaluation  is  con- 
ducted at  a test  site.  This  discussion  identifies  some  which  the  author 
considers  significant  and  should  be  considered  when  planning  and  conducting 
the  evaluation  program. 

The  test  and  evaluation  is  often  conducted  as  a combined  DT&E  and 
lOT&E  effort.  In  most  cases,  particularly  in  the  early  stages  of  test  and 
evaluation  during  the  validation  and  full  scale  development  phases,  DT&E 
will  take  priority  over  lOT&E  (1:28). 

Air  Force  personnel  responsible  for  accomplishing  the  qualitative 
maintainability  evaluation  will  be  representatives  of  the  developing 
consiand  and  Air  Force  Test  and  Evaluation  Center  (AFTEC)  test  team  personnel 
representing  the  using  conmand,  supporting  command,  and  training  conmand. 

In  recognition  of  the  fact  that  the  evaluation  program  will  often  be  a 
combined  DT&E  and  lOT&E  effort,  DT&E  accomplished  by  the  developing  command 
personnel  and  lOT&E  accomplished  by  the  AFTEC  test  team  personnel  must  be 
well  coordinated  (22:14).  Contractor  personnel  will  also  be  performing 
qualitative  maintainability  evaluations  as  a part  of  their  contractural 
maintainability  evaluation  responsibilities. 

In  almost  every  case  the  contractor  will  be  re.-jponsible  for  maintaining 
the  aircraft.  Depending  upon  the  arrangements  that  have  been  agreed  upon 
between  the  Air  Force  and  the  contractor.  Air  Force  maintenance  personnel 
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may  participate  in  maintenance  activities  in  an  observer  or  hands-un 
capacity.  The  maintenance  activities  can  include  both  on-aircraft  and 
off-aircraft  maintenance  (9:23). 

There  may  not  be  preliminary  Air  Force  type  technical  data  available. 
The  only  maintenance  data  available  may  be  contractor  engineering  drawing 
type  data  which  has  little  resemblance  to  Air  Force  type  technical  data 
(2:15). 

Only  limited  operational  support  and  test  equipment  may  be  available 
in  the  flight  test  program.  Some  of  this  equipment  may  be  government 
furnished  equipment  that  has  been  qualified  for  use  with  other  weapon 
systems  but  has  not  yet  been  qualified  for  the  aircraft  or  equipment  being 
tested.  Some  may  be  contractor  furnished  equipment  (CFE)  which  is  almost 
like  or  functionally  like  the  equipment  which  the  contractor  is  proposing 
be  used  in  the  operational  environment.  In  many  cases^  much  of  the  ei.,uipment 
will  be  CFE  which  only  has  application  for  the  flight  test  program  (2: 15) . 

There  are  almost  certain  to  be  numerous  differences  between  the 
configuration  of  the  test  aircraft  and  the  production  aircraft.  In  most 
cases  the  test  aircraft  will  contain  a large  amount  of  flight  test  instru- 
mentation hardware  and  wiring  that  Interferes  with  the  accomplishment  of 
maintenance  (6:G-13). 

Maintenance  will  generally  not  be  performed  in  the  flight  test  environ- 
mer _ in  the  same  manner  it  is  projected  to  be  accomplished  in  the  operation- 
al environment.  If  little  maintenance  technical  data  is  available,  many 
discrepancies  may  be  corrected  through  a process  of  a test  engineer 
determining  the  disposition  of  a discrepancy  and  working  with  the 
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appropriate  malatenance  personnel  to  fault  Isolate  and  correct  the 
discrepancy.  Many  maintenance  tasks  such  as  corrective  maintenance, 
scheduled  maintenance,  and  inspections  may  extend  over  prolonged  periods 
of  time  (several  days) , in  lieu  of  being  accomplished  as  single  uninter- 
rupted maintenance  tasks  as  they  might  be  performed  in  the  operational 
eiitf^ironment  (6:10). 

Maintainability  demonstrations  are  often  planned  as  a part  of  the 
qualitative  maintainability  evaluation  program.  However,  in  many  cases 
scheduling  the  inspections  and  accomplishing  them  is  difficult  because  of 
real  or  perceived  problems  of  the  demonstration  interfering  with  what  are 
determined  to  be  more  important  aspects  of  accomplishing  the  overall  test 
program;  for  :;xample,  getting  the  aircraft  repaired  expeditiously  in  order 
to  fly  it  again  soon  in  lieu  of  accomplishing  a scheduled  maintainability 
demonstration  (9:23). 

These  are  only  a few  of  the  factors  which  define  the  test  and  evalua- 
tion environment  in  which  a qualitative  maintainability  evaluation  may  be 
conducted.  However,  they  do  define  the  environment  sufficiently  enough  to 
permit  a better  understanding  of  the  significance  of  the  individual 
qualitative  maintainability  evaluation  factors  that  are  identified  in  the 
discussion  that  follows. 

Individual  Evaluation  Factors 

Many  people  might  be  inclined  to  say  that  there  are  an  endless  nimiber 
of  factors  one  should  consider  when  planning  and  conducting  a qualitative 
maintainability  evaluation  at  a test  site.  The  author  tends  to  agree. 
However,  in  this  study  project  he  attempts  to  identify  and  explore  those 

which  he  considers  to  be  most  significant. 
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When  considering  the  factors,  it  is  appropriate  to  identify  them  in 
two  different  categories.  One  category  deals  with  evaluation  factors  one 
should  consider  when  "planning  and  conducting"  the  evaluation.  These  might 
be  thought  of  as  administrative  factors.  The  other  category  deals  with  the 
evaluation  factors  one  should  consider  for  actually  "evaluating"  the  air- 
craft weapon  system. 

Many  of  the  factors  in  the  first  category  are  the  same  or  are  closely 
related  to  the  previously  identified  factors  which  define  the  environment 
in  which  test  and  evaluation  is  conducted  at  a test  site.  Both  planners  and 
conductors  of  qualitative  maintainability  test  and  evaluation  programs  must 
give  proper  attention  to  these  environmental  factors.  To  recapitulate,  the 
eight  environmental  factors  discussed  were:  1)  combined  DT&E  and  I0T6eE, 

2)  composition  of  the  maintenance  evaluation  personnel,  3)  contractor/Air 
Force  observer  or  hands-on  performance  of  maintenance,  4)  type  of  technical 
data,  5)  type  of  support  and  test  eqiiipment,  6)  aircraft  and  equipment 
configuration,  7)  methods  of  performing  maintenance  in  the  flight  test 
environment  compared  to  the  methods  projected  for  use  in  the  operational 
environment,  and  8)  accomplishing  maintainability  demonstrations. 

There  are  additional  factors  one  should  consider  in  "planning  and 
conducting"  the  evaluation.  The  number  and  qualifications  of  individual 
maintainability  evaluation  personnel  needed  to  accomplish  the  evaluation 
must  be  determined.  Individual  personnel  and  total  evaluation  team 
personnel  capabilities  must  be  assessed  in  order  to  effectively  plan  a 
realistic  evaluation  program  level  of  effort. 
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Contractor  data  systema  that  can  be  used  as  sources  of  msintcinabllity 
evaluation  data  must  be  Identified  and  plans  made  to  obtain  necessary  data 
as  appropriate  (3:3-7). 

Air  Force  data  collection,  processing,  and  evaluation  systems  should 
be  identified  or  developed  for  use  in  the  evaluation  program  (18:13). 

Planning  activities  r^hould  Include  insuring  the  availability  of 
contractor  and  Air  Force  maintenance  planning  documents  for  use  in  the 
evaluation  program.  These  5.nclude  such  documents  as  integrated  logistics 
support  plans,  maintenance  plans,  using  command  concepts  of  employment, 
maintenance  engineering  analyses,  an^  optimum  repair  level  analyses 
(19:5). 

Determinations  should  be  made  regarding  the  desirability  of  having  test 
site  evaluation  personnel  participate  in  off  site  contractor  and  Air  Force 
weapon  system  reviews.  If  so,  provisions  for  their  participation  should  be 
Included  in  appropriate  planning  documents  (5:8-5). 

Determinations  should  be  made  regarding  the  desirability  and  capability 
of  test  team  personnel  to  evaluate  and  report  on,  in  total  or  in  part, 
qualitative  maintainability  factors  whose  data  and/or  results  are  collected 
or  determined  at  locations  other  than  the  test  site. 

Evaluation  planning  should  Include  determinations  of  the  methods  that 
will  be  used  to  report  the  qualitative  maintainability  evaluation  results, 
who  will  receive  the  reports,  and  the  frequency  of  reporting. 

As  defined  earlier,  a second  category  of  evaluation  factors  deals  with 
the  evaluation  factors  one  should  consider  for  actually  "evaluating"  the 
aircraft  weapons  system.  This  involves  evaluating  both  on  and  off-aircraft 
equipment.  The  following  list  identifies  some  of  the  more  significant 


qualitative  malfltainabllity  evaluation  factors. 

( f 

1.  Evaluation  of  the  maintenance  plan  and  the  adequacy  of  the  main- 
tenance elements  of  the  support  concept  as  applicalile  (6:G-2). 

2.  Ground  servicing  provisions  and  procedures. 

3.  Equipment  Installation  provisions  and  procedures. 

4.  Inspection  requirements. 

5.  Repair  level  analyses. 

6.  Software  maintainability. 

7.  Self  sufficiencjT  capability. 

8.  Accessibility  of  parts,  test  points,  adjustments,  and  connections. 

9.  Test  instrumentation  effects  on  end  item  performance. 

10.  Accessibility/adaptability  for  in  flight  maintenance. 

11.  Design-dictated  or  permitted  preventive  maintenance  actions. 

12.  Proliferation  of  special  maintenance  tools. 

13.  Susceptibility  to  incorrect  maintenance  actions. 

14.  Susceptability  to  damage. 

15.  Adequacy  of  safety  and  protective  equipment. 

16.  Adequacy  of  standardization  and  Interchangeability  features. 

17.  Adequacy  of  security. 

18.  Adequacy  of  fall- safe  design  features. 

19.  Adequacy  of  corrosion  prevention  and  control. 

20.  Adequacy  of  time  change  procedures. 

21.  Adequacy  of  post-maintenance  operational  checks. 


22.  Special  handling  criteria. 

23.  Size  and  weight  limitation  criteria. 


24.  Environmental  extreme  criteria. 
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25.  Weapons?  load  criteria. 

26.  Alignment  and  calibration  criteria. 

27.  Cyclic  operation/duration  limits  criteria. 
2b.  Mission  variation  criteria. 

29.  Human  factors  engineering  (7:4-251). 


SECTION  III 

ANALYSIS  AND  EVALUATION  OF  II©IVLDUAL 
EVALUATION  FACTORS 

In  th  preceding  section,  44  Individual  evaluation  factors  to  consider 
when  planning  and  conducting  an  Air  Force  qualitative  maintainability  test 
and  evaluation  at  a test  site  were  Identified.  Fifteen  of  them  dealt 
specifically  with  the  evaluation  factors  one  should  consider  when  "plarmlng 
and  conducting"  the  evaluation;  29  of  them  dealt  with  the  evaluation  factors 
one  should  consider  for  actually  "evaluating"  the  aircraft  weapon  system. 

It  Is  not  the  author's  Intent  to  individually  report  the  analysis  and 
evaluation  of  each  of  the  44  individual  evaluation  factors.  In  the  dis- 
cussion that  follows  the  author  analyzes  and  evaluates  all  of  the  factors 
collectively,  stressing  points  regarding  individual  factors  he  deems  warrant 
special  connent. 

One  thing  that  is  common  to  all  of  the  evaluation  factors  is  the  fact 
that  they  are  factors  which  determine  the  quality  and  thoroughness  of  the 
qualitative  maintainability  evaluation  that  can  be  accomplished  at  a test 
site.  Some  of  the  factors  represent  greater  challenges  than  others  for 
personnel  responsible  for  planning  and  conducting  the  evaluation  program. 
Certainly  proper  recognition  of  the  test  environment  is  fundamental  in 
identifying  significant  factors  which  should  be  considered. 

An  analysis  and  evaluation  of  all  the  factors  highlights  the  Importance 
of  proper  planning  In  dealing  with  each  of  them  (10:5).  Many  of  the  factors 
are  related  and  the  planning  considerations  for  accommodating  them  are 
likewise  related.  One  of  the  most  important  considerations  in  dealing  with 
each  factor  Individually  or  several  related  ones  as  a group  is  to  begin  the 
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planning  effort  early.  This  effort  involves  early  identification  of  the 
total  qualitative  maintainability  evaluation  requirements,  effective 
communication  and  coordination  with  all  activities  involved  in  the  test 
planning  effort,  and  thorough  documentation  of  all  planning  factors  in  test 
planning  documents. 

All  of  the  individual  evaluation  factors  are  significant;  however,  the 
analysis  and  evaluation  of  all  the  factors  tends  to  indicate  the  15  factors 
one  should  consider  when  "planning  and  conducting"  the  evaluation  are  of 
greater  significance  than  the  29  factors  one  should  consider  for  actually 
"evaluating"  the  aircraft  weapon  system.  The  reason  is  that  the  quality  of 
the  evaluation  accomplished  in  terms  of  the  29  "evaluating"  factors  is 
significantly  dependent  upon  the  overall  quality  of  the  qualitative  main- 
tainability evaluation  program  planned  and  conducted  in  terms  of  the 
attention  given  to  the  15  "planning  and  conducting"  factors. 

The  most  important  aspect  regarding  the  15  factors  which  define  the 
environment  in  which  a qualitative  maintainability  test  and  evaluation  is 
conducted  is  to  insure  that  test  personnel  purposefully  consider  each 
applicable  factor  when  planning  and  conducting  the  evaluation. 

With  proper  planning,  factors  relating  to  the  combined  accomplishment 
of  DT&E  and  lOT&E  should  not  present  any  unsurmountable  obstacles  to 
effective  accomplishment  of  both  DT&E  and  lOT&E.  lOT&E  personnel  should  be 
able  to  accomplish  their  evaluation  with  no  danger  of  compromising  the 
required  independency  of  the  AFTEC  evaluation  (31:50).  In  order  to 
capitalize  on  the  advantages  of  Air  Force  personnel  performing  hands-on 
maintenance  with  the  contractor  maintenance  personnel,  planning  should 
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include  provisions  for  adequate  training  by  the  contractor  (2:16).  In  some 
cases  certification  by  the  contractor  of  the  Air  Force  maintentnce  per- 
sonnel's qualifications  to  work  on  the  weapon  system  hardware  is  a desirable 
objective  to  pursue.  Joint  Air  Force  and  contractor  performed  maintenance 
must  be  viewed  in  terms  of  the  impact  on  the  overall  flight  test  program. 

Areas  of  concern  include  such  things  as  the  Impact  of  maintenance  per- 
formed by  Air  Force  personnel  on  safety,  weapon  system  performance,  quality 
of  maintenance,  adherence  to  schedules,  etc. 

The  major  element  involved  with  the  four  environmental  factors  of 
preliminary  technical  data,  limited  operational  support  and  test  equipment, 
prototype  configuration  of  the  weapon  system,  and  the  method  of  performing 
maintenance  in  the  flight  test  environment  versus  the  operational  .iviron- 
ment,  is  the  fact  that  a major  portion  of  the  qualitative  maintainability 
evaluation  results  are  dependent  upon  the  judgement  of  experienced,  well 
trained  Air  Force  maintenance  personnel.  They  must  be  able  to  accurately 
extrapolate  their  observations  and  findings  from  the  flight  test  environment 
into  a meaningful  evaluation  of  the  weapon  system's  qualitative  maintain- 
ability in  the  operational  environment  (7:5-49). 

The  last  factor  which  defines  the  environment  in  which  an  evaluation 
is  conducted,  maintainability  demonstrations,  highlights  the  very  important 
issue  of  prior  planning  for  the  conduct  of  the  evaluation  program.  Decisions 
must  be  made  regarding  what  kind  of  maintenance  demonstration  programs  will 
be  implemented.  Programs  may  vary  considerably  in  scope.  Some  evaluators 
may  want  scheduled,  formal,  staged  demonstrations.  Others  ma>  elect  to 
wait  until  an  unscheduled  event  occurs  and  then  conduct  a formal  demonstra- 
tion. Others  may  be  willing  to  forego  formal  demonstrations  and  obtain 
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most  of  their  evaluation  findings  through  continuing  observer  and  hands-on 
maintenance  activities  with  the  contractor.  A key  element  in  developing 
a maintenance  demonstration  program  is  determining  which  maintenance  tasks 
will  be  demonstrated. 

The  remaining  "planning  and  conducting"  evaluation  factors  were  not 
identified  as  defining  the  environment  in  which  the  evaluation  is  conducted 
at  a teat  site.  They  included  evaluation  personnel  requirements,  contractor 
data  systems,  Air  Force  data  systems,  maintenance  planning  documents,  off  site 
participr.tion  by  evaluation  personnel  in  review  activities,  evaluation  of 
data  and  evaluation  results  comi.ng  from  off  site,  and  methods  of  reporting 
evaluation  results.  An  analysis  and  evaluation  of  these  factors  indicates 
that  effective  planning  and  firm  commitments  to  specific  management 
philosophies  are  absolutely  essential  in  effectively  accommodating  the 
factors  into  the  maintainability  evaluation  program.  These  factors  define 
an  area  of  many  management  decisions  and  failure  to  expend  an  appropriate 
level  of  early  effort  in  addressing  them  will  often  lead  to  a less  than 
satisfactory  evaluation  program.  Although  tne  significance  of  each  of 
these  factorr  should  be  readily  apparent,  it  is  appropriate  to  further 
comment  on  a few  of  them. 

In  developing  Air  Force  data  collection  systems  the  use  of  checklists, 
questionnaires,  and  fomatted  evaluation  sheets  may  often  be  appropriate 
for  gathering  qualitative  type  data.  Use  of  mechanized  data  systems  for 
processing  the  data  should  be  considered.  Opportunities  for  Air  Force 
maintenance  personnel  to  participate  in  flight  debriefings  often  provides 
vital  data  needed  in  evaluating  qualitative  maintainability. 
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Participation  by  evaluation  peraonnel  In  off  site  weapon  system  review 
activities  such  as  PDRs,  CDRs,  mockup  reviews,  etc.,  Is  often  very  profitable. 
It  permits  the  evaluation  personnel  to  gain  additional  knowledge  regarding 
the  weapon  system  design  and  configuration  and  also  permits  the  personnel 
to  make  meaningful  Inputs  to  the  reviews  as  appropriate. 

Careful  consideration  should  be  given  to  the  d(t termination  of  whether 
test  site  evaluation  personnel  will  evaluate  and  report  on  factors  whose 
data  and/or  results  are  collected  or  determined  at  locations  other  than  the 
test  site.  Factors  such  as  the  feasibility  of  sending  personnel  to  the 
off  site  locations  to  gather  the  data,  the  capability  of  the  personnel  to 
adequately  interpret  data  gathered  by  others  at  the  remote  locations,  the 
time  available  to  adequately  accomplish  the  on  site  evaluation  as  well  as 
the  off  site  evaluation,  etc.,  should  be  carefully  considered.  An  exeunple 
of  this  type  situation  is  requiring  test  site  personnel  to  evaluate  and 
report  on  the  qualitative  maintainability  of  the  weapon  system  at  the 
organizational.  Intermediate,  and  depot  levels  when  the  majority  of  their 
evaluation  effort  Is  devoted  to  organizational  level  maintenance  at  the 
test  site.  Also,  test  site  personnel  may  possess  the  skills  necessary  to 
evaluate  organizational  and  intermediate  level  maintainability,  but  not 
depot  level  maintainability. 

The  analysis  and  evaluation  of  the  29  evaluation  factors  one  should 
consider  for  actually  "evaluating"  the  aircraft  weapon  system  (pages  11-12) 
is  quite  straight  forward.  It  indicates  the  primary  emphasis  that  must 
be  directed  toward  accomplishing  all  of  the  individual  evaluations  is 
proper  planning  and  education  to  insure  evaluation  personnel  are  fully 
aware  of  the  factors  to  be  evaluated  and  can  relate  their  findings  to  the 
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qualitative  maintainability  of  the  aircraft  weapon  system  In  the  operational 
environment. 

Two  additional  Items  of  Interest  merit  connient  as  a result  of  the 
analysis  and  evaluation  of  the  44  qualitative  maintainability  factors. 

They  are  the  Interchange  of  qualitative  maintainability  Information  between 
Air  Force  and  contractor  evaluation  personnel  and  the  Interface  of  the 
qualitative  maintainability  evaluation  with  other  operational  suitability 
evaluation  factors. 

It  may  be  mutually  beneficial  to  the  Air  Force  nnd  contractors  if  an 
effective  program  of  qualitative  maintainability  Information  Interchange 
is  effected  at  the  test  site.  This  program  could  involve  Air  Force 
personnel  providing  contractor  personnel  with  evaluation  data,  both  positive 
features  and  discrepancies,  to  inform  contractors  of  the  Air  Force's 
findings.  Contractor  personnel  could  investigate  possible  solutions  to 
discrepancies  and  respond  to  the  Air  Force  regarding  possible  changes. 

In  this  manner  qualitative  maintainability  problems  could  be  Identified  at 
a very  low  level  and  possible  solutions  investigated  without  having  to 
elevate  the  problem  initially  to  a formal  Deficiency  Report  or  Engineering 
Change  Proposal  status  (16:14).  This  program  is  in  no  way  Intended  to 
circumvent  formal  contractural  requirements  regarding  the  submittal  of 
government  requests  to  contractors  for  accomplishment  of  work  or  Implemen- 
tation of  changes. 

The  second  item  of  Interest  deals  with  the  fact  that  the  qualitative 
maintainability  evaluation  will  interface  directly  with  other  operational 
suitability  evaluation  factors  and  In  many  cases  will  have  some  Impact  on 
them.  These  other  factors  include  reliability,  quantitative  maintainability. 
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availability,  and  logistica  aupportability . Qualitative  maintainability 
evaluation  personnel  must  be  constantly  alert  to  the  fact  that  these 
interfaces  may  be  potential  sources  of  tradeoffs  in  an  effort  to  achieve 
an  optifflvim  level  of  operational  suitability  (14:3). 
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SECTION  IV 


SUMMARY 

Conclusions 

The  DOD  Is  continuing  to  emphasize  the  Importance  of  realistic, 
effective,  and  meaningful  tests  and  evaluations  of  new  weapon  systems.  It 
Is  Important  that  test  and  evaluation  commence  as  early  as  possible  In  the 
system  acquisition  process,  and  an  estimate  of  the  military  utility  and 
the  operational  effectiveness  and  operational  suitability  Including  logistics 
support  requirements  be  accomplished  prior  to  making  a commitment  to  large- 
scale  production.  A key  element  in  a weapon  system's  DT&E  t.id  the  opera- 
tional suitability  portion  of  Its  OT&B  Is  an  evaluation  of  Its  qualitative 
mal ntalnablllty. 

The  author  Identifies  44  evaluation  factors  to  consider  when  planning 
and  conducting  an  Air  Force  aircraft  qualitative  maintainability  test  and 
evaluation  at  a test  site.  He  concludes  that  It  Is  appropriate  to  Identify 
them  In  two  different  categories.  One  category  deals  with  15  evaluation 
factors  one  should  consider  when  "planning  and  conducting"  an  evaluation. 

The  other  category  deals  with  29  evaluation  factors  one  should  consider 
for  actually  "evaluating"  the  aircraft  weapon  system. 

An  analysis  and  evaluation  of  the  evaluation  factors  reveals  the 
Importance  of  beginning  the  planning  effort  for  the  qualitative  maintain- 
ability evaluation  early  in  the  acquisition  program.  Also,  the  author 
concludes  that  the  quality  of  the  qualitative  maintainability  evaluation 
accomplished  in  terms  of  the  29  "evaluating"  factors  is  significantly 
dependent  upon  the  overall  quality  of  the  maintainability  evaluation 
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program  planned  and  conducted  In  terms  of  the  attention  given  to  the 
15  "planning  and  conducting"  factors.  The  44  Individual  evaluation  factors 
Identified  define  the  scope  of  the  evaluation  that  can  be  realistically 
accomplished  In  a test  and  evaluation  environment  at  a test  site. 

•i'; 

In  summary,  the  author  concludes  the  most  Important  aspect  regarding 
evaluation  factors  to  consider  when  planning  and  conducting  a qualitative 
maintainability  evaluation  is  to  identify  them  early  and  plan  the  evalua- 
tion program  effectively  using  the  factors  as  guides.  Once  planned, 
evaluation  managers  must  Insure  evaluation  personnel  are  fully  aware  of 
the  factors  to  be  evaluated  and  can  relate  their  findings  and  evaluation 
results  to  the  qualitative  maintainability  of  the  aircraft  weapon  system 
In  the  operational  environment. 

Recommendations 

The  author  recommends  the  study  of  effective  methods  to  accomplish 
Air  Force  aircraft  qualitative  maintainability  evaluations  at  test  sites 
continue.  More  Information  on  how  to  plan  and  conduct  evaluation 
programs  would  be  helpful  for  those  responsible  for  accomplishing  them. 

A need  exists  to  Increase  the  understanding  of  personnel  at  all  levels  In 
the  Air  Force  on  optimum  methods  to  accomplish  an  aircraft  qualitative 
maintainability  evaluation  at  a test  site. 

Implications 

The  implications  of  this  study  project  are  that  It  Identifies  factors 
which  the  author  considers  slgnlilcant  to  be  considered  In  planning  and 
conducting  an  aircraft  qualitative  maintainability  test  and  evaluation 
program  at  a test  site.  It  can  serve  as  an  aid  and  a source  of 
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